Managing and Aggregating Data Transfers in Data Centers

Distributed computing applications like MapReduce transfer massive amount of data between their successive processing stages. These data transfers, such as common shuffle and incast communication patterns, contribute most of the network traffic and thus have severe impacts on application performances. Despite such impacts, there has been relatively little work on decreasing the amount of traffic for computing such data transfers. We observe that the massive data flows in such a transfer already apply aggregate functions at the receiver side and the reduction in size between the input data and the output data is even pronounced. This motivates us to bring opportunities for performing interflow data aggregation during the transmission phase as early as possible rather than just at the receiver side. To this end, we first demonstrate the gain and feasibility of inter-flow data aggregation for data transfers in data centers with novel network structures. To achieve such a gain, such data transfers are normalized as the incast transfer. It is modeled as an incast minimal tree problem that is proved to be NP-hard in representative BCube and FBFLY data centers. We propose two approximate methods, the RS-based and ARS-based incast tree building methods, to generate an efficient incast tree based on only the labels of all incast members and the data center topology. We further present incremental methods to tackle the dynamic and fault-tolerant issues of the incast tree. Using a prototype implementation and large-scale simulations, we demonstrate that our method can significantly decrease the amount of network traffic, save the data center resources, and reduce the delay of the entire process of a job. Moreover, our proposals for BCube and FBFLY can also be applied to other novel data centers after minimal modifications.